Impact Evaluations and Development: Nonie Guidance on Impact Evaluation

In international development, impact evaluation is principally concerned with final results of interventions (programs, projects, policy measures, reforms) on the welfare of communities, households, and individuals, including taxpayers and voters. Impact evaluation is one tool within the larger toolkit of monitoring and evaluation (including broad program evaluations, process evaluations, ex ante studies, etc.).The Network of Networks for Impact Evaluation (NONIE) was established in 2006 to foster more and better impact evaluations by its membership -- the evaluation networks of bilateral and multilateral organizations focusing on development issues, as well as networks of developing country evaluators. NONIE's member networks conduct a broad set of evaluations, examining issues such as project and strategy performance, institutional development, and aid effectiveness. By sharing methodological approaches and promoting learning by doing on impact evaluations, NONIE aims to promote the use of this more specific approach by its members within their larger portfolio of evaluations. This document, by Frans Leeuw and Jos Vaessen, has been developed to support this focus.For development practitioners, impact evaluations play a keyrole in the drive for better evidence on results and development effectiveness. They are particularly well suited to answer important questions about whether development interventions do or do not work, whether they make a difference, and how cost-effective they are. Consequently, they can help ensure that scarce resources are allocated where they can have the most developmental impact

The Performance Paradox in the Public Sector

Administrative reform has led to a strong increase in the use of performance assessment instruments in the public sector. However, this has also led to several unintended consequences, such as the performance paradox, tunnel vision and ‘analysis paralysis’. These unintended consequences can reduce the quality of the knowledge about actual levels of performance, or even negatively affect performance. Examples can be found in all policy sectors. We argue that certain characteristics of the public sector – such as ambiguous policy objectives, discretionary authority of street level bureaucrats, simultaneous production and consumption of services, and the disjunction of costs and revenues – increase the risk of a performance paradox, either unintentionally or deliberately. Performance assessment should therefore take the special characteristics of the public sector into account and develop systems that can handle contested and multiple performance indicators, striking a balance in the degree of ‘meas

First inventory of policy on counterterrorism: Germany, France, Italy, Spain, the United Kingdom and the United States - 'research in progress'

This report presents a first inventory of measures aimed at combating radicalisation, extremism and terrorism (referred to in this report as counterterrorism policy) in Germany, France, Italy, Spain, the United Kingdom and the United States. Commissioned by the NCTb, the WODC (the Research and Documentation Centre of the Dutch Ministry of Justice) carried out a study into counterterrorism measures in place in the countries investigated in early 2006. This study comprises the first international inventory of a number of policy fields under development, whereby the counterterrorism policies discussed here are characterised by extremely rapid development and innovation. This inventory can serve as point of departure for the registration of policy developments and to obtain an insight into their effectiveness. The continuous monitoring of the developments is therefore recommended. Researchers in Germany, France, Italy, Spain and the United States have drawn up working documents, each with an inventory of the policy in place in that particular country, taking their information from public sources, mostly government publications and Websites, as well as articles, reports and books. Data on the United Kingdom were collected by the WODC. The closing date for the data collection was 1 February 2006. Besides the inventory, the researchers were invited: - to present a brief history of the development of the counterterrorism policy in the country in question, focusing on experiences with past terrorism and how it was combated - to devote attention where possible to (policy) assumptions on which more recent measures are based - to present available evaluations. Due to the study’s character of providing an inventory, differences may occur between the countries as regards topic detailing. This is “work in progress” involving a policy field in progress. There is no systematic comparison between the countries. It is not possible, therefore, to draw conclusions based on this report for the introduction of certain measures in the Netherlands. However, following a summary of the key study results, seven strategic issues were identified as they emerged from the material

Proportions of B-cell subsets are altered in incomplete systemic lupus erythematosus and correlate with interferon score and IgG levels

OBJECTIVES: Incomplete SLE (iSLE) patients display symptoms typical for SLE but have insufficient criteria to fulfil the diagnosis. Biomarkers are needed to identify iSLE patients that will progress to SLE. IFN type I activation, B-cell-activating factor (BAFF) and B-cell subset distortions play an important role in the pathogenesis of SLE. The aim of this cross-sectional study was to investigate whether B-cell subsets are altered in iSLE patients, and whether these alterations correlate with IFN scores and BAFF levels. METHODS: iSLE patients (n = 34), SLE patients (n = 41) with quiescent disease (SLEDAI ≤4) and healthy controls (n = 22) were included. Proportions of B-cell subsets were measured with flow cytometry, IFN scores with RT-PCR and BAFF levels with ELISA. RESULTS: Proportions of age-associated B-cells were elevated in iSLE patients compared with healthy controls and correlated with IgG levels. In iSLE patients, IFN scores and BAFF levels were significantly increased compared with healthy controls. Also, IFN scores correlated with proportions of switched memory B-cells, plasma cells and IgG levels, and correlated negatively with complement levels in iSLE patients. CONCLUSION: In this cross-sectional study, distortions in B-cell subsets were observed in iSLE patients and were correlated with IFN scores and IgG levels. Since these factors play an important role in the pathogenesis of SLE, iSLE patients with these distortions, high IFN scores, and high levels of IgG and BAFF may be at risk for progression to SLE

The effect of microcredit on women's control over household spending in developing countries: a systematic review

Background: Over the past three decades, microfinance activities have spread across the globe, reaching tens of millions of poor households with tailored financial services. Microfinance can best be described as a field of intervention rather than a particular instrument. Initially, microfinance usually meant microcredit for working capital and very small investments, but increasingly it has been broadened to include savings/deposits, a limited range of micro-insurance and payment services (including micro-leasing) as well as a somewhat broader range of credit products for more substantial investments. In this study we focused on microcredit activities, constituting the bulk of microfinance activities across the globe. Microcredit activities have affected the lives of clients and others in multiple ways. The most frequently reported types of effects of credit at individual, enterprise and household level are the following: income, expenditure smoothing, and poverty alleviation effects; business growth and employment effects; schooling effects; and effects in terms of women's empowerment. Despite the diversity in microcredit schemes, many share two characteristics: they target poor women and often rely on some type of group-based lending. Women's empowerment in relation to microcredit has been studied extensively within the context of this type of microcredit scheme. Most of these studies have been carried out in the context of microcredit group schemes in South Asia. It has been argued that access to microcredit can foster changes in individual attitudes of women (e.g. increased self-reliance), power relations within the household (e.g. control over resources) and social status. An important dimension of empowerment concerns women's control over household spending. The main assumption is that by providing credit to poor women, their direct control over expenditures within the household increases, with subsequent implications for the status of women and the well-being of women and other household members. Women's control over household spending is a frequently recurring aspect analyzed within the context of microcredit interventions, which allows us to study whether microcredit targeted at women affects women's control over household spending decisions and the circumstances in which this occurs. Despite the central and recurrent role across studies of this aspect of women's empowerment in relation to microcredit activities, there has been no previous review on this topic. The growing importance of microcredit has resulted in a vast number of research and evaluation studies, including impact studies. Consequently, the microfinance literature harbors a substantial number of synthesis studies which discuss a set of microcredit interventions and aim to generate overall conclusions on their effects. However, most of these studies face limitations in terms of depth of empirical assessment and the extent to which the identified effects can be attributed to microcredit. Moreover, methodological principles regarding comprehensive searches and principles of selection, coding, extraction and aggregation are often lacking in review studies. Partial exceptions are three recent systematic reviews which all differ in scope from the present one (Stewart et al., 2010; Duvendack et al. 2011; Stewart et al., 2012). The reviews respectively focus on microfinance (credit and savings) in Sub-Sahara Africa, microcredit worldwide, and microfinance worldwide (credit, saving and leasing). Overall, these reviews suggest that the effects of microcredit on women's empowerment are at best mixed. In part this can be explained by the heterogeneity in microcredit interventions, contexts and target groups. However, the existing reviews did not use statistical meta-analysis to synthesise evidence of effects, nor context-mechanism-outcome synthesis to understand the variation in effects. Objectives: The main objective of this study was to provide a systematic review of the evidence on the effects of microcredit on women's control over household spending in developing countries. More specifically, we aimed to answer two related research questions: 1) what does the impact evaluative evidence say about the causal relationship between microcredit and specific dimensions of women's empowerment (women'ss control over household spending); and 2) what are the mechanisms which mediate this relationship. We prioritise depth of analysis over breadth, thus the scope of this review is narrower than previous systematic reviews on microfinance (Stewart et al., 2010; Duvendack et al. 2011; Stewart et al., 2012). We focused on specific aspects of women's empowerment which allowed us to combine statistical meta-analysis and realist (context-mechanism-outcome) synthesis. Criteria for considering studies for this review: We included studies that analyzed the effects of microcredit schemes targeting poor women in low and middle income countries, as defined by the World Bank. Studies that did not include analysis on microcredit and the effect on one or more dimensions (specified in main body of the report) of women's control over household expenditures were excluded. Studies which gave evidence of addressing the attribution problem either through randomised design, quasi-experimental matching, or regression analysis, were included. In practice, women's control over household spending (as a key dimension of empowerment) is influenced by many different factors. By focusing on those studies which explicitly addressed the challenge of separating the effect of microcredit from other influencing factors, we developed what we consider to be the most credible evidence base for drawing conclusions about the effects of microcredit on women's control over household expenditures in different contexts. SEARCH STRATEGY We conducted a comprehensive search covering all relevant academic databases, internet search engines and web sites with published and unpublished research, and also carried out extensive manual searches of books and additional journals not included in electronic data bases (searches were concluded on December 31, 2011). We used back-referencing from recent studies as well as citation-tracking to identify additional relevant studies. Finally, authors of studies which we were unable to retrieve were contacted. In addition, we contacted experts on microcredit and women's empowerment for additional references which we might have missed. Search strategies in databases and journals were adapted for each source. Where possible we used the existing keyword indices of particular databases. In addition, we applied our own list of combinations of keywords covering all relevant terms relating to the independent variable (i.e. credit and its variations) and the dependent variable (i.e. dimensions of women's control over household spending, empowerment). Data collection and analysis: From the different searches we identified an initial number of 310 papers that were selected for full text examination. Eventually, 29 papers were retained for further analysis, corresponding to 25 unique studies. These 25 independent findings were included in the synthesis. However, based on a systematic risk of bias assessment we found that more than half of the included studies had high threats to internal validity. Moreover, only about half of the studies show a clear and coherent link between a theoretical framework on microcredit and women's control over household spending and empirical data analysis. It should be noted that reviewing and synthesizing quantitative results from studies is only one side of the coin. The other side is to understand what makes them work, or what prevents them from working. Consequently, we conducted a qualitative synthesis of the included studies, which focused on identifying the mechanisms which underlie the causal relationship between microcredit and women's control over household spending. RESULTS The results of the meta-analysis indicated that the effect sizes from experimental studies examining effects of microcredit on women's control over household spending are not statistically significantly different from zero. The effects from quasi-experimental studies are statistically insignificant overall, and at best of small magnitude for those studies assessed of being of high risk of bias. We conclude that there is no consistent evidence for an effect of microcredit on women's control over household spending. In the qualitative analysis, using Coleman's (1986, 1990) typology of mechanisms, we identified five different situational mechanisms and eight different action-formation mechanisms. Due to the combination of substantial heterogeneity in contexts (e.g. existing gender relations) and interventions (e.g. microcredit versus microcredit and additional services), and the lack of information in the studies on this heterogeneity, it was not possible to go beyond the identification of mechanisms, in terms of generating empirically tested articulated theories of change which are representative beyond a specific study context. Authors' conclusions: In line with three recent other reviews on microfinance (Stewart et al., 2010; Duvendack et al., 2011; Stewart et al. 2012) we found that the microcredit evidence base is extensive, yet most studies are weak methodologically. From those studies deemed comparable and of minimum acceptable quality, we concluded that overall there is no evidence for an effect of microcredit on women's control over household spending. Women's control over household resources constitutes an important intermediary dimension in processes of women's empowerment. Given the overall lack of evidence for an effect of microcredit on women's control over household resources it is therefore very unlikely that, overall, microcredit has a meaningful and substantial impact on empowerment processes in a broader sense. While impacts on empowerment may appear to have occurred in particular studies, the high risk of bias of studies providing positive assessments suggests that such findings are of limited validity. Our conclusions on the effects of microcredit on empowerment are also in line with previous systematic reviews by Duvendack et al. (2011) and Stewart (et al. 2010) who report to a limited extent on empowerment effects. Consequently, there appears to be a gap between the often optimistic societal belief in the capacity of microcredit to ameliorate the position of women in decision-making processes within the household on the one hand, and the empirical evidence base on the other hand. However, our review markedly differs from previous reviews in two regards. First, we specifically focused on microcredit and women's empowerment captured through women's control over household expenditures. Second, as a result of this narrower focus, we were able to conduct statistical meta-analysis and extract behavioral mechanisms which can help to explain why and how microcredit can make a difference. The advantage of our approach was that the identified mechanisms all stem from studies which show evidence of addressing the attribution problem. Consequently, we can be quite confident of the insights that they provided on the effects of microcredit on women's control over household spending for particular populations of microcredit female clients and their families. Those studies that showed evidence of addressing the attribution problem were relatively weak on underlying theory. Moreover, they often lacked essential information such as the nature of the intervention and how it related to empowerment (e.g. how solidarity groups affect empowerment processes) or the slowly evolving gender relations in different contexts (e.g. the evolution of societal norms and the relationship with power relations in the household). A next logical step would be to undertake a systematic review of qualitative studies which often provide rich and context-specific information on microcredit and women's decision-making power in the household. Such a review should ideally build on the mechanisms identified in the present review and would bring us closer to uncovering credible theories of microcredit and the circumstances in which it may change women's decision-making power

The politicisation of evaluation: constructing and contesting EU policy performance

Although systematic policy evaluation has been conducted for decades and has been growing strongly within the European Union (EU) institutions and in the member states, it remains largely underexplored in political science literatures. Extant work in political science and public policy typically focuses on elements such as agenda setting, policy shaping, decision making, or implementation rather than evaluation. Although individual pieces of research on evaluation in the EU have started to emerge, most often regarding policy “effectiveness” (one criterion among many in evaluation), a more structured approach is currently missing. This special issue aims to address this gap in political science by focusing on four key focal points: evaluation institutions (including rules and cultures), evaluation actors and interests (including competencies, power, roles and tasks), evaluation design (including research methods and theories, and their impact on policy design and legislation), and finally, evaluation purpose and use (including the relationships between discourse and scientific evidence, political attitudes and strategic use). The special issue considers how each of these elements contributes to an evolving governance system in the EU, where evaluation is playing an increasingly important role in decision making